Add picker tracing decorator to scheduling pipeline#1708
Conversation
github-actions
Bot
added
the
size/L
| logger.V(logutil.VERBOSE).Info("Running picker plugin", "plugin", p.picker.TypedName()) | ||
| logger.V(logutil.DEBUG).Info("Candidate pods for picking", "endpoints-weighted-score", scoredEndpoints) | ||
|
|
||
| ctx, span := tracing.Tracer(schedplugins.TracerScope).Start(ctx, "pick_endpoints", |
There was a problem hiding this comment.
TraceScope should be llm-d-router/pkg/epp/scheduling
There was a problem hiding this comment.
Hi @gyliu513 Can you please review again?
Wrap the picker call in runPickerPlugin in a single inline "pick_endpoints" span
via tracing.Tracer(schedplugins.TracerScope) (SpanKindInternal), recording the
candidate and selected endpoint counts (llm_d.epp.picker.candidate_endpoints,
llm_d.epp.picker.selected_endpoints = len(result.TargetEndpoints), nil-guarded)
plus the conditional gen_ai.request.{model,id} keys. request is threaded into
runPickerPlugin (sole caller Run) so the span carries the request keys, matching
the filter and scorer spans. Follows the single-step span convention from llm-d#1565
and llm-d#1693; picking behavior and the per-plugin latency metric are unchanged.
Refs: llm-d#1694, llm-d#1483
Signed-off-by: ChethanUK <chethanuk@outlook.com>
de9d2fc to
59c4b6a
Compare
|
looks good, thanks @chethanuk ! @ahg-g can you help review? Thanks |
| if result != nil { | ||
| selected = len(result.TargetEndpoints) | ||
| } | ||
| span.SetAttributes(attribute.Int("llm_d.epp.picker.selected_endpoints", selected)) |
There was a problem hiding this comment.
I am not sure if it is useful to track the number of selected endpoints, in most cases it will be one with a fallback. I think what is more useful is to record the scores of the top 5 endpoints.
The picker almost always returns a single target (with an occasional fallback), so the count of selected endpoints carries little diagnostic signal. What explains a routing decision is the score distribution across the strongest candidates: how close the runner-up was, and how the top scores were spread. Replace the selected_endpoints count attribute with the names and weighted scores of the highest-scoring candidates, ordered by descending score and capped at five so the payload stays bounded on production fleet sizes (~100 pods). The top set is captured before Pick runs because pickers reorder the candidate slice in place. Signed-off-by: ChethanUK <chethanuk@outlook.com>
Signed-off-by: chethanuk <chethanuk@outlook.com>
|
lgtm, thanks @chethanuk ! @ahg-g can you help approve and merge If you are ok with the fix, thanks! |
|
@chethanuk please rebase to pick up the mardown link changes from main. Other than that (and waiting on checks to pass) - lgtm: the implementation follows the existing pick_pd_profile/score_prefix_cache span convention and addresses the earlier feedback from @ahg-g about tracking top-5 scores instead of endpoint count. Tests cover span nesting, capping, and nil-result cases well. Leaving final approval to @ahg-g - good to go from my point of view. |
|
Rebased - Please merge thanks |
4 participants
What
Add a single OTel
pick_endpointsspan over the scheduling picker stage,carrying the candidate/selected endpoint counts and request-correlation keys.
Implements #1694 (sub-task of #1483).
Design
Inline, single-step span, following the merged tracing convention from #1565
(the repo standard:
pick_pd_profile/score_prefix_cacheare traced inline,not via a decorator) and the maintainer guidance on #1693:
runPickerPluginstarts onepick_endpointsspan (SpanKindInternal) viatracing.Tracer(schedplugins.TracerScope), withdefer span.End()so it endson every path including a nil result.
llm_d.epp.picker.candidate_endpoints(scored-endpoint count),llm_d.epp.picker.selected_endpoints(len(result.TargetEndpoints),nil-guarded), plus the conditional shared
gen_ai.request.model/gen_ai.request.idkeys — matchingscore_prefix_cache. Notype/nameattrs (the span name identifies the operation).
requestis threaded intorunPickerPlugin(sole callerRun) so the spancarries the request keys, consistent with the filter and scorer spans. The span
context is threaded into
picker.Pick(...), so inner spans nest. Pickingbehavior and the per-plugin latency metric are unchanged.
Scope:
schedplugins.TracerScope=llm-d-router/pkg/epp/framework/plugins/scheduling.Review feedback addressed
TracedPickerdecorator is removed entirely; there is no per-request wrapper.
otel.Tracer()(gemini): now usestracing.Tracer(...), sospans carry the BuildRef / commit-sha instrumentation metadata.
new test comments capture only non-obvious rationale.
decision (single-step span,
llm_d.epp.picker.*keys, package scope).Tests
scheduler_profile_picker_tracing_test.go(spans read as a slice via atracetestrecorder): single selected (candidate 3 -> selected 1) withname/kind/parent and counts; multiple selected (== 2); nil result -> span ended,
selected_endpoints == 0, no panic;gen_ai.*omitted when request fields areempty; inner delegate span nests under
pick_endpoints.Gates:
go build ./...,go test ./pkg/epp/scheduling/... -race,go vet,and
make lint(new-only) all green.Refs: #1694, #1483